Detecting Road Conditions Based on Braking Event Data Received from Vehicles

ABSTRACT

Data is received regarding vehicle braking events, each event occurring on one of a plurality of vehicles, and each event associated with a location. A determination is made that the braking events correspond to a pattern. Based on determining that the braking events correspond to the pattern, a first location is identified. In response to identifying the first location, at least one action is performed.

RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/010,060 filed Jun. 15, 2018, the entiredisclosure of which application is hereby incorporated herein byreference.

This application is related to U.S. Pat. No. 10,997,429 issued on May 4,2021, Attorney Docket No. 120426-020700/US (2018-0207.00/US), entitled“Determining Autonomous Vehicle Status Based on Mapping of CrowdsourcedObject Data,” by Gil Golov, the entire contents of which application isincorporated by reference as if fully set forth herein.

This application is also related to U.S. Pat. No. 10,894,545 issued onJan. 19, 2021, entitled “Configuration of a Vehicle Based on CollectedUser Data,” by Robert Richard Noel Bielby, the entire contents of whichapplication is incorporated by reference as if fully set forth herein.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to monitoringoperating events for vehicles in general, and more particularly, but notlimited to, monitoring data regarding braking events occurring onvehicles.

BACKGROUND

A user of a vehicle can be a driver in the case of a manually-drivenvehicle. In other cases, such as for an autonomous vehicle, the user ofthe vehicle typically performs fewer control actions than a “driver” asregards the operation of the vehicle. For example, in some cases, theuser may simply select a destination to which the vehicle travels, butwithout performing any directional or other control of the immediatemovement of the vehicle on the roadway.

Recent developments in the technological area of autonomous drivingallow a computing system to operate, at least under some conditions,control elements of a vehicle without the assistance from a user of thevehicle. For example, sensors (e.g., cameras and radars) can beinstalled on a vehicle to detect the conditions of the surroundings ofthe vehicle on a roadway. One function of these sensors is to detectobjects that are encountered during travel of the vehicle.

Autonomous vehicles use a variety of sensors and artificial intelligencealgorithms to detect these objects and to analyze the changingenvironment around the vehicle during travel. Objects that areencountered may include, for example, traffic lights, road signs, roadlanes, etc. Failing to detect certain of these objects could cause anunexpected or undesired behavior of the vehicle, and in some cases couldexpose passengers of the vehicle and/or others outside of the vehicle(e.g., in the immediate area surrounding the vehicle) to danger.

In some cases, an object may be positioned in a way that creates anunsafe driving condition (e.g., a deep pothole in the center of a road).Failure by a driver or an autonomous vehicle navigation system to detectthe unsafe condition may create a physical danger of injury to thedriver and/or other passengers of a vehicle (e.g., a vehicle thatsuddenly encounters a deep pothole or other unsafe road conditionwithout warning).

During normal operation of a vehicle, the various sensors are used tooperate the vehicle. For example, a computing system installed on thevehicle analyzes the sensor inputs to identify the conditions andgenerate control signals or commands for the autonomous adjustments ofthe direction and/or speed of the vehicle, without any input from ahuman operator of the vehicle. Autonomous driving and/or advanced driverassistance system (ADAS) typically involves an artificial neural network(ANN) for the identification of events and/or objects that are capturedin sensor inputs.

In general, an artificial neural network (ANN) uses a network of neuronsto process inputs to the network and to generate outputs from thenetwork. Each neuron m in the network receives a set of inputs p_(k),where k=1, 2, . . . , n. In general, some of the inputs to a neuron maybe the outputs of certain neurons in the network; and some of the inputsto a neuron may be the inputs to the network as a whole. Theinput/output relations among the neurons in the network represent theneuron connectivity in the network.

Each neuron m has a bias b_(m), an activation function f_(m), and a setof synaptic weights w_(mk) for its inputs p_(k) respectively, where k=1,2, . . . , n. The activation function may be in the form of a stepfunction, a linear function, a log-sigmoid function, etc. Differentneurons in the network may have different activation functions.

Each neuron m generates a weighted sum s_(m) of its inputs and its bias,where s_(m)=b_(m)+w_(m1)×p₁+w_(m2)×p₂+ . . . +w_(mn)×p_(n). The outputa_(m) of the neuron m is the activation function of the weighted sum,where a_(m)=f_(m) (s_(m)).

The relations between the input(s) and the output(s) of an ANN ingeneral are defined by an ANN model that includes the data representingthe connectivity of the neurons in the network, as well as the biasb_(m), activation function f_(m), and synaptic weights w_(mk) of eachneuron m. Using a given ANN model, a computing device computes theoutput(s) of the network from a given set of inputs to the network.

For example, the inputs to an ANN network may be generated based oncamera inputs; and the outputs from the ANN network may be theidentification of an item, such as an event or an object.

For example, U.S. Pat. App. Pub. No. 2017/0293808, entitled“Vision-Based Rain Detection using Deep Learning”, discloses a method ofusing a camera installed on a vehicle to determine, via an ANN model,whether the vehicle is in rain or no rain weather.

For example, U.S. Pat. App. Pub. No. 2017/0242436, entitled “RoadConstruction Detection Systems and Methods”, discloses a method ofdetecting road construction using an ANN model.

For example, U.S. Pat. Nos. 9,672,734 and 9,245,188 discuss techniquesfor lane detection for human drivers and/or autonomous vehicle drivingsystems.

In general, an ANN may be trained using a supervised method where thesynaptic weights are adjusted to minimize or reduce the error betweenknown outputs resulted from respective inputs and computed outputsgenerated from applying the inputs to the ANN. Examples of supervisedlearning/training methods include reinforcement learning, and learningwith error correction.

Alternatively or in combination, an ANN may be trained using anunsupervised method where the exact outputs resulted from a given set ofinputs is not known a priori before the completion of the training. TheANN can be trained to classify an item into a plurality of categories,or data points into clusters.

Multiple training algorithms are typically employed for a sophisticatedmachine learning/training paradigm.

The disclosures of the above discussed patent documents are herebyincorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 illustrates a system to receive data regarding braking eventsoccurring on a plurality of vehicles, according to one embodiment.

FIG. 2 shows an example of a vehicle configured using an ArtificialNeural Network (ANN) model, according to one embodiment.

FIG. 3 shows a method to identify a location based on determining thatbraking events correspond to a pattern, according to one embodiment.

FIG. 4 shows a method to identify a location based on analysis of dataregarding braking events for a plurality of vehicles, according to oneembodiment.

FIG. 5 shows an autonomous vehicle controlled and/or configured inresponse to determining identifying a location associated with brakingevents, according to one embodiment.

FIG. 6 shows a vehicle controlled and/or configured via a communicationinterface using a cloud service, according to one embodiment.

FIG. 7 is a block diagram of an autonomous vehicle including one or morevarious components and/or subsystems, each of which can be updated invarious embodiments to configure the vehicle and/or perform otheractions associated with the vehicle.

FIG. 8 is a block diagram of a centralized autonomous vehicle operationssystem, according to various embodiments.

DETAILED DESCRIPTION

Currently, the technology supporting autonomous and other vehiclescontinues to improve. Improvements in digital camera technology, lightdetection and ranging (LIDAR), and other technologies have enabledvehicles to navigate roadways independent of drivers or with limitedassistance from drivers. In some environments, such as factories,autonomous vehicles operate without any human intervention whatsoever.

While autonomous technology is primarily focused on controlling themovement of vehicles in a traditional sense, little emphasis has beenplaced on alternative applications that may be implemented on top ofthese autonomous systems. Indeed, application-level systems generallytend to reinforce existing uses of autonomous systems. For example,experimental uses of autonomous technology have been utilized to performfunctions such as returning vehicles to a known location afterdelivering a passenger or performing refueling of vehicles while notutilized by passengers.

However, these approaches fail to fully utilize the hardware andprocessing power being implemented in autonomous vehicles, or in othervehicles utilizing automated driver assistance systems. Thus, therecurrently exists a need in the state of the art of autonomous and othervehicles to provide additional services leveraging the existing hardwareinstalled within such vehicles.

In particular, there is a need to solve the technical problem ofdetecting unsafe road conditions that may be encountered by anautonomous or other vehicle during its operation. In particular, thistechnical problem includes the need to determine whether navigationand/or other control of the vehicle should be performed to avoid theunsafe condition. In many cases, this detection of unsafe roadconditions needs to be determined in real-time.

At least some embodiments disclosed herein relate to monitoring dataregarding braking events occurring on vehicles. The data is receivedfrom each of the vehicles (e.g., received by a server). Analysis of thebraking event data is used to identify unsafe locations, and in responseto perform control actions for a vehicle currently being operated (e.g.,controlling navigation of a current vehicle based on analysis of thebraking event data). These embodiments provide a technological solutionto the above technical problem by analyzing data regarding brakingevents received from other vehicles to identify the unsafe location andcause a control response for a current vehicle. Based on this analysis,an unsafe location on a road (e.g., corresponding to an unsafe roadcondition) is identified and/or determined. In response, acurrently-operating vehicle (sometimes referred to herein as simply“current” vehicle) is, for example, alerted and/or controlled in orderto avoid the unsafe road condition. The current vehicle is, for example,a manually-driven vehicle or an autonomous vehicle (e.g., a car, truck,aircraft, drone, watercraft, etc.). In one example, an unsafe roadcondition may include alien objects that are unsafely positioned on aroad. For example, a tree may have unexpectedly fallen on a road due toa recent storm, and the tree is blocking safe travel on the road.

In one embodiment, a cloud service (sometimes referred to as simply thecloud) is used to detect unsafe driving road conditions. For example,when a driver performs an emergency braking, or the automatic emergencybraking system of a vehicle is activated, the location of that brakingevent is transmitted to, for example, a server or other computing devicein the cloud service. The location itself may be determined, forexample, based on location data (e.g., geographic coordinates) providedfrom the vehicle itself (e.g., by a GPS location system) and/or locationdata otherwise associated with or known about the vehicle. The cloudsearches for a pattern of emergency braking at the same location (withina tolerance such as, for example, a predetermined distance), which havebeen reported by numerous vehicles (e.g., other vehicles traveling thesame road prior to the current vehicle). This pattern may be used tomake a determination that driving is unsafe in that particular location.

In one embodiment, braking event data received from vehicles is analyzedto determine that the braking events correspond to a pattern. Forexample, pattern recognition can be used on the received data. In onecase, machine learning is used to recognize patterns or regularities indata. In some cases, a pattern recognition system can be trained fromlabeled training data (e.g., supervised learning). In other cases, whenno labeled data is available, other algorithms can be used to identifypreviously unknown patterns (e.g., unsupervised learning).

In one embodiment, a braking event is identified based on a measurementof a brake pedal in a vehicle. For example, a braking event may beidentified based on a foot pressure or extent of movement as compared toa predetermined threshold. In another embodiment, a braking event isidentified based on a rate of deceleration of the vehicle. For example,if a rate of deceleration exceeds a predetermined threshold, then abraking event is identified. In another example, the rate ofdeceleration is observed over a predetermined time period (e.g., therate is averaged over the time period).

In one embodiment, braking event data is analyzed to identify a patternin which two or more locations of vehicles experiencing a braking eventfall within a predetermined distance of another location (e.g.,associated with collected or received data), or a predetermined region(e.g., a circular or other area of a predetermined radius or otherdimension that encloses locations of braking events).

In one embodiment, identifying an unsafe location requires that a numberof braking events within a time period exceed a predetermined number orthreshold. For example, the identification can require that three ormore braking events occur in a given day.

In one embodiment, in response to identifying an unsafe location, atleast one action is performed. For example, a communication can be sentto a current vehicle that identifies the location. In another example,the location can be transmitted to another computing device that is usedto control traffic flows, vehicle navigation, etc. In another example,this information can be used, for example, to improve road conditions,traffic sign and traffic lights infrastructure around the unsafelocation (e.g., this may improve driving safety).

In another embodiment, braking event data is received by a server (e.g.,a server of a cloud service) and stored as a map of locations of thebraking events. Data for each braking event is, for example, receivedfrom other vehicles that have previously experienced a braking event(e.g., the hardware of these other vehicles is used to collect sensorand other data regarding the braking event that occurred during travel).For example, these other vehicles can be vehicles that have previouslytraveled over the same road that a current vehicle is presentlytraveling on. By storing data regarding the braking events (e.g.,storing in a map), the current vehicle can be controlled duringoperation so as to avoid any unsafe locations that are determined fromthe braking event data.

In one embodiment, additional data from other vehicles can be receivedand/or stored that relates to objects detected by the other vehicles.For example, a fallen tree may be detected within a predetermined timeof the occurrence of a braking event. For example, the map above canstore data regarding the stop sign detected by one or more priorvehicles. The map includes a location of the stop sign along with dataregarding an associated braking event.

Data received from a current vehicle traveling at or near this samelocation is compared to data stored in the map. In one example, based oncomparing the data received from the current vehicle to the stored mapdata, navigation or another operating status of the current vehicle iscontrolled or changed (e.g., an updated configuration is performed). Forexample, it may be determined that the current vehicle should beginbraking at least a predetermined distance or time prior to reaching alocation that has been identified as unsafe.

In one embodiment, a cloud service receives braking event data fromnumerous vehicles. The braking event data is used to create a map ofunsafe locations. These unsafe locations are identified based onanalysis of the braking event data. The map data can be provided, forexample, as an online service to other computing devices associated withvehicle operation. For example, the map data can be used by a serverthat controls operation of one or more autonomous vehicles. In anotherexample, the service can transmit data to one or more vehicles that isused to control at least one action performed by the vehicle.

As mentioned above, in addition to braking event data, vehicles may sendother data. For example, data regarding physical objects detected byvehicles can be sent to a server in a cloud. A map is stored thatincludes locations for each of these detected objects (e.g., along withbraking event data). For example, the map can include data collected bythe prior vehicles. For example, locations of physical objects can bebased on data received from the prior vehicles.

In one embodiment, objects detected by prior vehicles (e.g., passiveobjects, such as traffic signs, traffic lights, etc.) are transmitted tothe cloud service. The cloud service creates a dynamic map containingthe type of object detected and its location (e.g., the map stores datathat a stop sign is located at a position x, y). The cloud servicestores the map (e.g. in a database or other data repository). Brakingevent data is also stored in the map (e.g., braking event locationsassociated with objects within a predetermined distance of the brakingevent location).

In one embodiment, in response identifying an unsafe location (e.g., asdetermined based on pattern recognition using braking event data), aserver can perform one or more actions. For example, the server can senda communication to the current vehicle. In one case, the communicationcan cause the current vehicle to take corrective actions, such asterminating an autonomous navigation mode, braking, or changing course.

In one embodiment, in response to receiving a communication from aserver, a current vehicle can switch off its autonomous driving mode,use a backup system, and/or activate a braking system to stop thevehicle.

In another embodiment, the cloud service can send a communication to aserver or other computing device that monitors an operating status forother vehicles (e.g., a central monitoring service). For example, thecloud service can send a communication to a server operated bygovernmental authorities. The communication can, for example, identifythat a road has an unsafe condition (e.g., at an identified location).In some cases, in response to a determination that the current vehiclehas been in an accident associated with an identified unsafe location,the communication can be sent to the server or other computing device.In such a case, one or more indications provided to the server or othercomputing device can include data obtained from the current or anothervehicle associated with a braking event at the accident location (e.g.,data stored by the vehicle regarding operating functions and/or state ofthe vehicle prior to the accident, such as within a predetermined timeperiod prior to the accident).

In one embodiment, the determination whether a vehicle has experienced abraking event and/or been involved in an accident can be based on datafrom one or more sensors of the vehicle. For example, data from anaccelerometer of the vehicle can indicate a rapid deceleration of thevehicle (e.g., deceleration exceeding a threshold). In another case,data can indicate that an emergency system of the vehicle has beenactivated, such as for example, an airbag, an emergency braking system,etc. In some embodiments, any one and/or a combination of the foregoingevents can be deemed to be a braking event for which location data istransmitted to a server.

In one embodiment, a route (e.g., data for the current location of thevehicle) taken by a vehicle being monitored is sent periodically to acloud service. One or more sensors on the current vehicle are used toobtain data regarding braking events and/or objects in the environmentof the vehicle as it travels along the route. Data from the sensorsand/or data generated based on analysis of sensor data and/or other datacan be, for example, transmitted to the cloud service wirelessly (e.g.,using a 3G, 4G, or 5G network or other radio-based communicationsystem).

In one embodiment, in response to identifying an unsafe road locationdetermined based on braking event data, one or more actions of a vehicleare configured. For example, an over-the-air firmware update can be sentto the vehicle for updating firmware of a computing device of thevehicle (e.g., this update causes the vehicle to avoid identified unsafelocations and/or objects at such locations). In one example, thefirmware updates a navigation system of the vehicle. The updatedconfiguration is based at least in part on analysis of data that iscollected from other vehicles.

In various other embodiments, the configuration of one or more actionsperformed by a vehicle in response to identifying a location mayinclude, for example, actions related to operation of the vehicle itselfand/or operation of other system components mounted in the vehicleand/or otherwise attached to the vehicle. For example, the actions mayinclude actions implemented via controls of an infotainment system, awindow status, a seat position, and/or driving style of the vehicle.

In some embodiments, the analysis of braking event and/or sensor datacollected by the current or other prior vehicles includes providing thedata as an input to a machine learning model. The current vehicle iscontrolled by performing one or more actions that are based on an outputfrom the machine learning model.

In one example, a machine learning model is trained and/or otherwiseused to configure a vehicle (e.g., tailor actions of the vehicle). Forexample, the machine learning model may be based on pattern matching inwhich prior patterns of sensor inputs or other data is correlated withdesired characteristics or configuration(s) for operation of thevehicle.

In one embodiment, data received from the current vehicle may includesensor data collected by the vehicle during its real world services(e.g., when the user is a driver or a passenger). In one embodiment, thedata is transmitted from the vehicles to a centralized server (e.g., ofa cloud service), which performs machine learning/training, using asupervised method and the received sensor data and/or other data, togenerate an updated ANN model that can be subsequently loaded into thevehicle to replace its previously-installed ANN model. The model is usedto configure the operation of the vehicle.

In some embodiments, the driver can take over certain operations fromthe vehicle in response to the vehicle receiving a communication that anunsafe location has been identified. One or more cameras of the vehicle,for example, can be used to collect image data that assists inimplementing this action. In one example, the vehicle is configured inreal-time to respond to the received braking event and/or object data.

FIG. 1 illustrates a system to receive data regarding braking eventsoccurring on a plurality of vehicles, according to one embodiment. Thesystem uses an Artificial Neural Network (ANN) model in someembodiments. The system of FIG. 1 includes a centralized server 101 incommunication with a set of vehicles 111, . . . , 113 via acommunication network 102.

In one embodiment, data regarding braking events occurring on vehicles(e.g., other or prior vehicle 113) is received by server 101 viacommunication network 102. The received data includes a location foreach of the braking events. For example, the received data can includean event location 163 for each braking event 161. Braking event 161 caninclude data such as, for example, an identifier, a type of brakingevent, etc. The received braking event data can be stored as part of mapdata 160.

In some embodiments, additional data is received by server 101 from thevehicles. This can include, for example, data regarding detected objectssuch as object type 162 and object location 164. This additional datacan be stored as part of map data 160. Also, additional data such assensor data 103 can be received from the vehicles.

The braking event data received from the vehicles by server 101 isanalyzed. For example, this analysis can include pattern recognition orother data analysis (e.g., determining a correlation of braking eventdata to other data) to determine that the braking events correspond to apattern. For example, event location data 163 can be analyzed to detecta pattern. In one embodiment, this pattern detection can be based atleast in part on an output from artificial neural network model 119.

Based on analysis of the received braking event data, a location isidentified (e.g., an unsafe road obstacle). For example, server 101 maydetermine that a set of braking events corresponds to a pattern and acorresponding location is identified based on this determination. In oneexample, a location can be determined as being unsafe based on numerousemergency braking activations on vehicles at that location or within apredetermined distance of the identified location.

In response to identifying the location, at least one action isperformed. For example, server 101 can transmit a communication tocurrent vehicle 111 that causes the vehicle to change a navigation pathand/or activate a braking system when within a predetermined distance ofthe identified unsafe location.

In some embodiments, in addition to sending data regarding brakingevents, vehicle 113 and/or other prior vehicles send data regardingobjects detected during travel (e.g. vehicle 113 can be traveling priorto current vehicle 111, which arrives later at the same location wherean object has been detected by vehicle 113). These objects can include,for example, object 155 and object 157. Sensors of vehicle 113 and theother prior vehicles collect and/or generate data regarding the objectsthat have been detected. Data regarding detected objects can be analyzedin conjunction with braking event data in order to identify a locationthat prompts an action.

Data regarding the detected objects is sent, via communications network102, to a computing device such as server 101 (e.g., which may be partof a cloud service). Server 101 receives the object data from vehicle113 and the other prior vehicles. Server 101 stores a map (e.g.,including map data 160), which may include a number of records for eachobject. In one example, map data 160 includes object type 162 and objectlocation 164 for each object. Map data 160 also may include brakingevent data 161 and corresponding event locations 163, as mentionedabove.

Subsequent to receiving the data regarding detected objects from theprior vehicles, a current vehicle 111 also may transmit data regardingnew objects that are being detected during travel. For example, object155 can be a new object from the perspective of vehicle 111.

Server 101 receives data regarding object 155 from vehicle 111. In someembodiments, server 101 may determine, based on comparing the dataregarding object 155 that is received from vehicle 111 to data regardingobject 155 that is stored in map data 160, how vehicle 111 shouldrespond to an unsafe location that has been identified.

In some cases, vehicle 111 sends its current location to server 101. Thelocation of vehicle 111 is compared to object location 164 for object155. Server 101 determines a response that vehicle 111 should performassociated with object 155.

In one embodiment, in response to identifying a location, server 101performs one or more actions. For example, server 101 can transmit acommunication to vehicle 111 that causes a termination of an autonomousdriving mode.

In one embodiment, sensor data 103 can be collected in addition to mapdata 160. Sensor data 103 can be, for example, provided by the currentvehicle 111 and/or prior vehicles 113 (e.g., sensor data 103 may be fordata other than object data, such as temperature, acceleration, audio,etc.). Sensor data 103 can be used in combination with map data 160and/or other new data received from current vehicle 111 to perform ananalysis of received data (including received braking event data). Insome cases, some or all of the foregoing data can be used to trainartificial neural network model 119. Additionally, in some cases, anoutput from artificial neural network model 119 can be used as part ofmaking a determination of an unsafe location.

In some embodiments, at least a portion of map data 160 can betransmitted to vehicle 111 and a determination or control action (e.g.,navigation path change) regarding an operating status of vehicle 111 canbe locally determined by a computing device mounted on or within vehicle111. In some embodiments, artificial neural network model 119 itselfand/or associated data can be transmitted to and implemented on vehicle111 and/or other vehicles. An output from artificial neural networkmodel 119 can be used to determine actions performed in response toidentifying an unsafe location based on braking event data (e.g., anidentification received from server 101 by vehicle 111).

In one embodiment, data from vehicle 111 (or from vehicle 113) can becollected by sensors located in the vehicle. The collected data isanalyzed, for example, using a computer model such as an artificialneural network (ANN) model. In one embodiment, the collected data isprovided as an input to the ANN model. For example, the ANN model can beexecuted on server 101 and/or vehicle 111. The vehicle 111 is controlledbased on at least one output from the ANN model. For example, thiscontrol includes performing one or more actions based on the output.These actions can include, for example, control of steering, braking,acceleration, and/or control of other systems of vehicle 111 such as aninfotainment system and/or communication device.

In one embodiment, the server 101 includes a supervised training module117 to train, generate, and update ANN model 119 that includes neuronbiases 121, synaptic weights 123, and activation functions 125 ofneurons in a network used for processing braking event data, and/orother collected data regarding a vehicle and/or sensor data generated inthe vehicles 111, . . . , 113.

In one embodiment, once the ANN model 119 is trained and implemented(e.g., for autonomous driving and/or an advanced driver assistancesystem), the ANN model 119 can be deployed on one or more of vehicles111, . . . , 113 for usage.

In various embodiments, the ANN model is trained using data as discussedabove. The training can be performed on a server and/or the vehicle.Configuration for an ANN model as used in a vehicle can be updated basedon the training. The training can be performed in some cases while thevehicle is being operated.

Typically, the vehicles 111, . . . , 113 have sensors, such as a visiblelight camera, an infrared camera, a LIDAR, a RADAR, a sonar, and/or aset of peripheral sensors. The sensors of the vehicles 111, . . . , 113generate sensor inputs for the ANN model 119 in autonomous drivingand/or advanced driver assistance system to generate operatinginstructions, such as steering, braking, accelerating, driving, alerts,emergency response, etc.

During the operations of the vehicles 111, . . . , 113 in theirrespective service environments, the vehicles 111, . . . , 113 encounteritems, such as events or objects, that are captured in the sensor data.The ANN model 119 is used by the vehicles 111, . . . , 113 to providethe identifications of the items to facilitate the generation ofcommands for the operations of the vehicles 111, . . . , 113, such asfor autonomous driving and/or for advanced driver assistance. Capturingof certain of this data can be triggered in response to determining thata braking event has or is occurring. For example, vehicle 113 maydetermine that a braking event is occurring and activate collection ofpredetermined types or extent of sensor data (e.g., image data fordetected objects).

In one example, a vehicle may communicate, via a wireless connection 115to an access point (or base station) 105, with the server 101 to submitthe sensor input to enrich the sensor data 103 as an additional datasetfor machine learning implemented using the supervised training module117. The wireless connection 115 may be made via a wireless local areanetwork, a cellular communications network, and/or a communication link107 to a satellite 109 or a communication balloon. In one example, userdata collected from a vehicle can be similarly transmitted to theserver.

Optionally, the sensor input stored in the vehicle may be transferred toanother computer for uploading to the centralized server 101. Forexample, the sensor input can be transferred to another computer via amemory device, such as a Universal Serial Bus (USB) drive, and/or via awired computer connection, a Bluetooth or WiFi connection, a diagnosistool, etc.

Periodically, the server 101 may run the supervised training module 117to update the ANN model 119 based on updated data that has beenreceived. The server 101 may use the sensor data 103 enhanced with theother data based on prior operation by similar vehicles (e.g., brakingevent data received from vehicle 113) that are operated in the samegeographical region or in geographical regions having similar trafficconditions (e.g., to generate a customized version of the ANN model 119for the vehicle 111).

Optionally, the server 101 uses the sensor data 103 along with objectdata received from a general population of vehicles (e.g., 111, 113) togenerate an updated version of the ANN model 119. The updated ANN model119 can be downloaded to the current vehicle (e.g., vehicle 111) via thecommunications network 102, the access point (or base station) 105, andcommunication links 115 and/or 107 as an over-the-air update of thefirmware/software of the vehicle.

Optionally, the vehicle 111 has a self-learning capability. After anextended period on the road, the vehicle 111 may generate a new set ofsynaptic weights 123, neuron biases 121, activation functions 125,and/or neuron connectivity for the ANN model 119 installed in thevehicle 111 using the sensor inputs it collected and stored in thevehicle 111. As an example, the centralized server 101 may be operatedby a factory, a producer or maker of the vehicles 111, . . . , 113, or avendor of the autonomous driving and/or advanced driver assistancesystem for vehicles 111, . . . , 113.

FIG. 2 shows an example of a vehicle configured using an ArtificialNeural Network (ANN) model, according to one embodiment. The vehicle 111of FIG. 2 includes an infotainment system 149, a communication device139, one or more sensors 137, and a computer 131 that is connected tosome controls of the vehicle 111, such as a steering control 141 for thedirection of the vehicle 111, a braking control 143 for stopping of thevehicle 111, an acceleration control 145 for the speed of the vehicle111, etc. In some embodiments, vehicle 113 of FIG. 1 has a similarconfiguration and/or similar components.

The computer 131 of the vehicle 111 includes one or more processors 133,memory 135 storing firmware (or software) 127, the ANN model 119 (e.g.,as illustrated in FIG. 1), and other data 129.

In one example, firmware 127 is updated by an over-the-air update inresponse to a communication from server 101 sent in response toidentifying an unsafe location (e.g., located on a road that vehicle 111is travelling on). Alternatively, and/or additionally, other firmware ofvarious computing devices or systems of vehicle 111 can be updated.

The one or more sensors 137 may include a visible light camera, aninfrared camera, a LIDAR, RADAR, or sonar system, and/or peripheralsensors, which are configured to provide sensor input to the computer131. A module of the firmware (or software) 127 executed in theprocessor(s) 133 applies the sensor input to an ANN defined by the model119 to generate an output that identifies or classifies an event orobject captured in the sensor input, such as an image or video clip.Data from this identification and/or classification can be included inobject data sent from a vehicle to server 101 as discussed above.

Alternatively, and/or additionally, the identification of an unsafelocation and/or classification of a braking event or object generated bythe ANN model 119 can be used by an autonomous driving module of thefirmware (or software) 127, or an advanced driver assistance system, togenerate a response. The response may be a command to activate and/oradjust one of the vehicle controls 141, 143, and 145. In one embodiment,the response is an action performed by the vehicle where the action hasbeen configured based on an update command from server 101 (e.g., theupdate command can be generated by server 101 in response to determiningthat vehicle 111 is approaching a location identified based on analysisof braking event data). In one embodiment, prior to generating thecontrol response, the vehicle is configured. In one embodiment, theconfiguration of the vehicle is performed by updating firmware ofvehicle 111. In one embodiment, the configuration of the vehicleincludes updating of the computer model stored in vehicle 111 (e.g., ANNmodel 119).

The server 101 stores the received sensor input as part of the sensordata 103 for the subsequent further training or updating of the ANNmodel 119 using the supervised training module 117. When an updatedversion of the ANN model 119 is available in the server 101, the vehicle111 may use the communication device 139 to download the updated ANNmodel 119 for installation in the memory 135 and/or for the replacementof the previously installed ANN model 119. These actions may beperformed in response to determining that vehicle 111 is failing toproperly detect objects and/or in response to identifying an unsafelocation.

In one example, the outputs of the ANN model 119 can be used to control(e.g., 141, 143, 145) the acceleration of a vehicle (e.g., 111), thespeed of the vehicle 111, and/or the direction of the vehicle 111,during autonomous driving or provision of advanced driver assistance.

Typically, when the ANN model is generated, at least a portion of thesynaptic weights 123 of some of the neurons in the network is updated.The update may also adjust some neuron biases 121 and/or change theactivation functions 125 of some neurons. In some instances, additionalneurons may be added in the network. In other instances, some neuronsmay be removed from the network.

In one example, data obtained from a sensor of a vehicle may be an imagethat captures an object using a camera that images using lights visibleto human eyes, or a camera that images using infrared lights, or asonar, radar, or LIDAR system. In one embodiment, image data obtainedfrom at least one sensor of the vehicle is part of the collected datafrom the vehicle that is analyzed. In some instances, the ANN model isconfigured for a particular vehicle based on the sensor and othercollected data.

FIG. 3 shows a method to identify a location based on determining thatbraking events correspond to a pattern, according to one embodiment. Inblock 601, data is received regarding braking events for vehicles (e.g.,vehicle 113 and/or other vehicles). Each braking event occurs on one ofthe vehicles, and each event is associated with a location at which thebraking event occurs.

In block 603, a determination is made that the braking events correspondto a pattern. The pattern may be, for example, that a predeterminednumber of the braking events occur within a predetermined distance ofone another.

In block 605, a first location is identified based on determining thatthe braking events correspond to the pattern. For example, a number ofbraking events occurring within a predetermined distance can beassociated with a location at which the events occurred. In someembodiments, this location can be specified as a defined area, a zone, aphysical region or territory, etc.

In block 607, in response to identifying the first location (e.g.,identifying that the first location is unsafe based on variousparameters), at least one action is performed. For example, the actioncan be performed by server 101 (e.g., sending of a communication tocurrent vehicle 111 to cause a control action on the vehicle). Inanother example, the action can include sending at least onecommunication to a computing device other than the new vehicle. In oneexample, the computing device is a server that monitors an operatingstatus for each of one or more vehicles.

In one embodiment, a system includes: at least one processor; and memorystoring instructions configured to instruct the at least one processorto: receive data regarding braking events, each event occurring on oneof a plurality of vehicles, and each event associated with a location;determine that the braking events correspond to a pattern; identify,based on determining that the braking events correspond to the pattern,a first location; and in response to identifying the first location,perform at least one action.

In one embodiment, determining that the braking events correspond to thepattern comprises comparing a deceleration of the respective vehicle foreach of the braking events to a predetermined threshold.

In one embodiment, determining that the braking events correspond to thepattern comprises at least one of: determining that the respectivelocation for each of the braking events is within a predetermineddistance of the first location; or determining that a distance betweenthe respective locations for the braking events is within apredetermined value.

In one embodiment, each of the braking events corresponds to activationof at least one of an automatic emergency braking system or an anti-lockbraking system of the respective vehicle.

In one embodiment, determining that the braking events correspond to thepattern comprises comparing a measurement associated with activation ofa braking system of the respective vehicle to a predetermined threshold.

In one embodiment, the measurement is associated with movement of abrake pedal by a user of the respective vehicle.

In one embodiment, the predetermined threshold is a level of pressure oran extent of motion associated with depressing of the brake pedal.

In one embodiment, determining that the braking events correspond to thepattern comprises comparing a number of the braking events that occurwithin a predetermined time period to a threshold.

In one embodiment, determining that the braking events correspond to thepattern comprises identifying the pattern based at least in part on anoutput from a machine learning model.

In one embodiment, the machine learning model is an artificial neuralnetwork.

In one embodiment, performing the at least one action comprises sendinga communication to a first vehicle, the communication causing the firstvehicle to perform at least one of controlling an operation of the firstvehicle based on an output from an artificial neural network,deactivating an autonomous driving mode of the first vehicle, orcontrolling a navigation system of the first vehicle.

In one embodiment, the instructions are further configured to instructthe at least one processor to store a map including the respectivelocations for each of the braking events.

In one embodiment, the instructions are further configured to instructthe at least one processor to provide, based on the map, a navigationservice to a first vehicle.

In one embodiment, the data regarding braking events is received by aserver that monitors a respective operating status for each of theplurality of vehicles, and monitoring the operating status includesreceiving data regarding objects detected by each of the vehicles.

In one embodiment, performing the at least one action is based on anoutput from a machine learning model, and the machine learning model istrained using training data, the training data comprising data collectedby sensors of the plurality of vehicles.

FIG. 4 shows a method to identify a location based on analysis of dataregarding braking events for a plurality of vehicles (e.g., vehicle 113and/or other vehicles), according to one embodiment. In block 611, datais received regarding braking events occurring on the vehicles. The dataincludes a location for each braking event.

In block 613, the received data is analyzed. For example, braking eventdata 161 and event location data 163 can be analyzed to determine thatthe data corresponds to a pattern and/or other correlation.

In block 615, based on analyzing the received data, a first location isidentified. For example, the location can be identified by geographiccoordinates and a type of condition associated with the result of theanalysis. This type of condition can, for example, be communicated to avehicle for use in determining a control action.

In block 617, in response to identifying the first location, one or moreactions are performed. These actions can be performed by, for example,server 101, server 301, or server 701.

In one embodiment, a method includes: receiving, by at least oneprocessor, data regarding braking events occurring on a plurality ofvehicles, the data including a location for each of the braking events;analyzing, by the at least one processor, the received data;identifying, based on analyzing the received data, a first location; andin response to identifying the first location, performing at least oneaction.

In one embodiment, performing the at least one action comprisesconfiguring, based on analyzing the received data, a first vehicle,wherein the first vehicle is an autonomous vehicle comprising acontroller and a storage device, wherein configuring the first vehiclecomprises updating firmware of the controller, and wherein the updatedfirmware is stored in the storage device.

In one embodiment, the method further comprises, in response toidentifying the first location, analyzing image data associated with thefirst location, wherein analyzing the image data comprises performingpattern recognition on image data to determine at least one objectassociated with the first location.

In one embodiment, analyzing the received data comprises determiningthat the respective locations for the braking events are within apredetermined area.

In one embodiment, a non-transitory computer storage medium storesinstructions which, when executed on a computing device, cause thecomputing device to perform a method comprising: receiving, by at leastone processor, data regarding braking events occurring on a plurality ofvehicles, the data including a location for each of the braking events;analyzing, by the at least one processor, the received data; and inresponse to analyzing the received data, performing at least one action.

FIG. 5 shows an autonomous vehicle 303 controlled and/or configured inresponse to identifying a location associated with braking events,according to one embodiment. A system controls a display device 308 orother device, system, or component of an autonomous vehicle 303. Forexample, a controller 307 controls the display of images on one or moredisplay devices 308. Controller 307 also controls navigation of thevehicle (e.g. using braking control 143 of FIG. 2).

Server 301 may store, for example, map data 160. Server 301 maydetermine, using map data 160, that vehicle 303 is approaching an unsafelocation (and/or is failing to properly detect objects). In response tothis determination, server 301 may cause the controller 307 to terminatean autonomous navigation mode. Other actions can be performed inresponse to this determination including, for example, configuring avehicle 303 by updating firmware 304, updating computer model 312,updating data in database 310, and/or updating training data 314.

The controller 307 may receive data collected by one or more sensors306. The sensors 306 may be, for example, mounted in the autonomousvehicle 303. The sensors 306 may include, for example, a camera, amicrophone, a motion detector, and/or a camera. At least a portion ofthe sensors may provide data associated with objects newly detected byvehicle 303 during travel.

The sensors 306 may provide various types of data for collection by thecontroller 307. For example, the collected data may include image datafrom the camera and/or audio data from the microphone.

In one embodiment, the controller 307 analyzes the collected data fromthe sensors 306. The analysis of the collected data includes providingsome or all of the collected data as one or more inputs to a computermodel 312. The computer model 312 can be, for example, an artificialneural network trained by deep learning. In one example, the computermodel is a machine learning model that is trained using training data314. The computer model 312 and/or the training data 314 can be stored,for example, in memory 309. An output from the computer model 312 can betransmitted to server 301 as part of object data for comparison to mapdata 160.

In one embodiment, memory 309 stores a database 310, which may includedata collected by sensors 306 and/or data received by a communicationinterface 305 from computing device, such as, for example, a server 301(server 301 can be, for example, server 101 of FIG. 1 in someembodiments). In one example, this communication may be used towirelessly transmit collected data from the sensors 306 to the server301. The received data may include configuration, training, and otherdata used to configure control of the display devices 308 or othercomponents by controller 307.

In one example, the received data may include data collected fromsensors of autonomous vehicles other than autonomous vehicle 303. Thisdata may be included, for example, in training data 314 for training ofthe computer model 312. The received data may also be used to update aconfiguration of a machine learning model stored in memory 309 ascomputer model 312.

In FIG. 5, firmware 304 controls, for example, the operations of thecontroller 307 in controlling the display devices 308 and othercomponents of vehicle 303. The controller 307 also can, for example, runthe firmware 304 to perform operations responsive to communications fromthe server 301. The autonomous vehicle 303 includes volatile DynamicRandom-Access Memory (DRAM) 311 for the storage of run-time data andinstructions used by the controller 307.

In one embodiment, memory 309 is implemented using variousmemory/storage technologies, such as NAND gate based flash memory,phase-change memory (PCM), magnetic memory (MRAM), resistiverandom-access memory, and 3D XPoint, such that the memory 309 isnon-volatile and can retain data stored therein without power for days,months, and/or years.

In one embodiment server 301 communicates with the communicationinterface 305 via a communication channel. In one embodiment, the server301 can be a computer having one or more Central Processing Units (CPUs)to which vehicles, such as the autonomous vehicle 303, may be connectedusing a computer network. For example, in some implementations, thecommunication channel between the server 301 and the communicationinterface 305 includes a computer network, such as a local area network,a wireless local area network, a cellular communications network, or abroadband high-speed always-connected wireless communication connection(e.g., a current or future generation of mobile network link).

In one embodiment, the controller 307 performs data intensive, in-memoryprocessing using data and/or instructions organized in memory 309 orotherwise organized in the autonomous vehicle 303. For example, thecontroller 307 can perform a real-time analysis of a set of datacollected and/or stored in the autonomous vehicle 303. In someembodiments, the set of data further includes collected or configurationupdate data obtained from server 301.

At least some embodiments of the systems and methods disclosed hereincan be implemented using computer instructions executed by thecontroller 307, such as the firmware 304. In some instances, hardwarecircuits can be used to implement at least some of the functions of thefirmware 304. The firmware 304 can be initially stored in non-volatilestorage media, such as by using memory 309, or another non-volatiledevice, and loaded into the volatile DRAM 311 and/or the in-processorcache memory for execution by the controller 307. In one example, thefirmware 104 can be configured to use the techniques discussed hereinfor controlling display or other devices of a vehicle as configuredbased on collected user data.

FIG. 6 shows a vehicle 703 controlled and/or configured via acommunication interface using a cloud service, according to oneembodiment. For example, vehicle 703 receives a control communicationand/or is configured in response to a determination by server 701 thatvehicle 703 is approaching an unsafe location identified, at least inpart, based on braking event data received from other vehicles.

The vehicle 703 includes a communication interface 705 used to receive aconfiguration update, which is based on analysis of collected objectdata. For example, the update can be received from server 701 and/orclient device 719. Communication amongst two or more of the vehicle 703,a server 701, and a client device 719 can be performed over a network715 (e.g., a wireless network). This communication is performed usingcommunication interface 705.

In one embodiment, the server 701 controls the loading of configurationdata (e.g., based on analysis of collected data) of the newconfiguration into the memory 709 of the vehicle. In one embodiment,data associated with usage of vehicle 703 is stored in a memory 721 ofclient device 719.

A controller 707 controls one or more operations of the vehicle 703. Forexample, controller 707 controls user data 714 stored in memory 709.Controller 707 also controls loading of updated configuration data intomemory 709 and/or other memory of the vehicle 703. Controller 707 alsocontrols display of information on display device(s) 708. Sensor(s) 706provide data regarding operation of the vehicle 703. At least a portionof this operational data can be communicated to the server 701 and/orthe client device 719.

Memory 709 can further include, for example, configuration data 712and/or database 710. Configuration data 712 can be, for example, dataassociated with operation of the vehicle 703 as provided by the server701. The configuration data 712 can be, for example, based on collectedand/or analyzed object data (including braking event data received byserver 701 from vehicles).

Database 710 can store, for example, configuration data for a userand/or data collected by sensors 706. Database 710 also can store, forexample, navigational maps and/or other data provided by the server 701.

In one embodiment, when a vehicle is being operated, data regardingobject detection activity of vehicle 703 can be communicated to server701. This activity may include navigational and/or other operationalaspects of the vehicle 703.

As illustrated in FIG. 6, controller 707 also may control the display ofimages on one or more display devices 708 (e.g., an alert to the usercan be displayed in response to determining by server 701 and/orcontroller 707 that vehicle 703 is failing to properly detect objects).Display device 708 can be a liquid crystal display. The controller 707may receive data collected by one or more sensors 706. The sensors 706may be, for example, mounted in the vehicle 703. The sensors 706 mayinclude, for example, a camera, a microphone, a motion detector, and/ora camera.

The sensors 706 may provide various types of data for collection and/oranalysis by the controller 707. For example, the collected data mayinclude image data from the camera and/or audio data from themicrophone. In one embodiment, the image data includes images of one ormore new objects encountered by vehicle 703 during travel.

In one embodiment, the controller 707 analyzes the collected data fromthe sensors 706. The analysis of the collected data includes providingsome or all of the object data to server 701.

In one embodiment, memory 709 stores database 710, which may includedata collected by sensors 706 and/or configuration data received bycommunication interface 705 from a computing device, such as, forexample, server 701. For example, this communication may be used towirelessly transmit collected data from the sensors 706 to the server701. The data received by the vehicle may include configuration or otherdata used to configure control of navigation, display, or other devicesby controller 707.

In FIG. 6, firmware 704 controls, for example, the operations of thecontroller 707. The controller 707 also can, for example, run thefirmware 704 to perform operations responsive to communications from theserver 701.

The vehicle 703 includes volatile Dynamic Random-Access Memory (DRAM)711 for the storage of run-time data and instructions used by thecontroller 707 to improve the computation performance of the controller707 and/or provide buffers for data transferred between the server 701and memory 709. DRAM 711 is volatile.

FIG. 7 is a block diagram of an autonomous vehicle including one or morevarious components and/or subsystems, each of which can be updated invarious embodiments to configure the vehicle and/or perform otheractions associated with the vehicle (e.g., configuration and/or otheractions performed in response to identifying a location based onanalyzing braking events of a plurality of vehicles). The systemillustrated in FIG. 7 may be installed entirely within a vehicle.

The system includes an autonomous vehicle subsystem 402. In theillustrated embodiment, autonomous vehicle subsystem 402 includes mapdatabase 402A, radar devices 402B, Lidar devices 402C, digital cameras402D, sonar devices 402E, GPS receivers 402F, and inertial measurementunits 402G. Each of the components of autonomous vehicle subsystem 402comprise standard components provided in most current autonomousvehicles. In one embodiment, map database 402A stores a plurality ofhigh-definition three-dimensional maps used for routing and navigation.Radar devices 402B, Lidar devices 402C, digital cameras 402D, sonardevices 402E, GPS receivers 402F, and inertial measurement units 402Gmay comprise various respective devices installed at various positionsthroughout the autonomous vehicle as known in the art. For example,these devices may be installed along the perimeter of an autonomousvehicle to provide location awareness, collision avoidance, and otherstandard autonomous vehicle functionality.

Vehicular subsystem 406 is additionally included within the system.Vehicular subsystem 406 includes various anti-lock braking systems 406A,engine control units 402B, and transmission control units 402C. Thesecomponents may be utilized to control the operation of the autonomousvehicle in response to the streaming data generated by autonomousvehicle subsystem 402A. The standard autonomous vehicle interactionsbetween autonomous vehicle subsystem 402 and vehicular subsystem 406 aregenerally known in the art and are not described in detail herein.

The processing side of the system includes one or more processors 410,short-term memory 412, an RF system 414, graphics processing units(GPUs) 416, long-term storage 418 and one or more interfaces 420.

The one or more processors 410 may comprise central processing units,FPGAs, or any range of processing devices needed to support theoperations of the autonomous vehicle. Memory 412 comprises DRAM or othersuitable volatile RAM for temporary storage of data required byprocessors 410. RF system 414 may comprise a cellular transceiver and/orsatellite transceiver. Long-term storage 418 may comprise one or morehigh-capacity solid-state drives (SSDs). In general, long-term storage418 may be utilized to store, for example, high-definition maps, routingdata, and any other data requiring permanent or semi-permanent storage.GPUs 416 may comprise one more high throughput GPU devices forprocessing data received from autonomous vehicle subsystem 402A.Finally, interfaces 420 may comprise various display units positionedwithin the autonomous vehicle (e.g., an in-dash screen).

The system additionally includes a reporting subsystem 404 whichperforms data collection (e.g., collection of data obtained from sensorsof the vehicle that is used to drive the vehicle). The reportingsubsystem 404 includes a sensor monitor 404A which is connected to bus408 and records sensor data transmitted on the bus 408 as well as anylog data transmitted on the bus. The reporting subsystem 404 mayadditionally include one or more endpoints to allow for systemcomponents to transmit log data directly to the reporting subsystem 404.

The reporting subsystem 404 additionally includes a packager 404B. Inone embodiment, packager 404B retrieves the data from the sensor monitor404A or endpoints and packages the raw data for transmission to acentral system (illustrated in FIG. 8). In some embodiments, packager404B may be configured to package data at periodic time intervals.Alternatively, or in conjunction with the foregoing, packager 404B maytransmit data in real-time and may compress data to facilitate real-timecommunications with a central system.

The reporting subsystem 404 additionally includes a batch processor404C. In one embodiment, the batch processor 404C is configured toperform any preprocessing on recorded data prior to transmittal. Forexample, batch processor 404C may perform compression operations on thedata prior to packaging by packager 404B. In another embodiment, batchprocessor 404C may be configured to filter the recorded data to removeextraneous data prior to packaging or transmittal. In anotherembodiment, batch processor 404C may be configured to perform datacleaning on the recorded data to conform the raw data to a formatsuitable for further processing by the central system.

Each of the devices is connected via a bus 408. In one embodiment, thebus 408 may comprise a controller area network (CAN) bus. In someembodiments, other bus types may be used (e.g., a FlexRay or MOST bus).Additionally, each subsystem may include one or more additional bussesto handle internal subsystem communications (e.g., LIN busses for lowerbandwidth communications).

FIG. 8 is a block diagram of a centralized autonomous vehicle operationssystem, according to various embodiments. As illustrated, the systemincludes a number of autonomous vehicles 502A-502E. In one embodiment,each autonomous vehicle may comprise an autonomous vehicle such as thatdepicted in FIG. 7. Each autonomous vehicle 502A-502E may communicatewith a central system 514 via a network 516. In one embodiment, network516 comprises a global network such as the Internet.

In one example, central system 514 is implemented using one or more ofservers 101, 301, and/or 701. In one example, one or more of autonomousvehicles 502A-502E are autonomous vehicle 703.

The system additionally includes a plurality of client devices 508A,508B. In the illustrated embodiment, client devices 508A, 508B maycomprise any personal computing device (e.g., a laptop, tablet, mobilephone, etc.). Client devices 508A, 508B may issue requests for data fromcentral system 514. In one embodiment, client devices 508A, 508Btransmit requests for data to support mobile applications or web pagedata, as described previously.

In one embodiment, central system 514 includes a plurality of servers504A. In one embodiment, servers 504A comprise a plurality of front endwebservers configured to serve responses to client device 508A, 508B.The servers 504A may additionally include one or more applicationservers configured to perform various operations to support one or morevehicles.

In one embodiment, central system 514 additionally includes a pluralityof models 504B. In one embodiment, models 504B may store one or moreneural networks for classifying autonomous vehicle objects. The models504B may additionally include models for predicting future events. Insome embodiments the models 504B may store a combination of neuralnetworks and other machine learning models.

Central system 514 additionally includes one or more databases 504C. Thedatabases 504C may include database record for vehicles 504D,personalities 504E, and raw data 504F. Raw data 504F may comprise anunstructured database for storing raw data received from sensors andlogs as discussed previously.

The present disclosure includes methods and apparatuses which performthe methods described above, including data processing systems whichperform these methods, and computer readable media containinginstructions which when executed on data processing systems cause thesystems to perform these methods.

Each of the server 101 and the computer 131 of a vehicle 111, . . . , or113 can be implemented as one or more data processing systems. A typicaldata processing system may include includes an inter-connect (e.g., busand system core logic), which interconnects a microprocessor(s) andmemory. The microprocessor is typically coupled to cache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. An apparatus comprising: at least one processor;and memory storing instructions configured to instruct the at least oneprocessor to: monitor a respective operating status for each of aplurality of autonomous vehicles, the monitoring including receivingdata regarding objects detected by the vehicles; determine that a firstvehicle of the vehicles is failing to properly detect objects whenoperating in an autonomous driving mode; and in response to determiningthat the first vehicle is failing to properly detect objects, send acommunication to the first vehicle, the communication causing the firstvehicle to deactivate the autonomous driving mode of the first vehicle.2. The apparatus of claim 1, wherein the communication further causesupdating firmware of a controller of the first vehicle.
 3. The apparatusof claim 1, wherein the first vehicle comprises a storage deviceconfigured to store an artificial neural network (ANN) model used todetect objects when operating in the autonomous driving mode.
 4. Theapparatus of claim 3, wherein the instructions are further configured toinstruct the at least one processor to: in response to determining thatthe first vehicle is failing to properly detect objects, send updatedfirmware to the first vehicle; wherein the updated firmware is stored inthe storage device, and the updated firmware is used by the firstvehicle to update the ANN model.
 5. The apparatus of claim 3, whereinthe ANN model is trained using training data, and the training datacomprises data collected by sensors of the autonomous vehicles.
 6. Theapparatus of claim 1, wherein the instructions are further configured toinstruct the at least one processor to: store a map based on dataregarding braking events received from the vehicles other than the firstvehicle; and provide, based on the map, a navigation service to thefirst vehicle.
 7. The apparatus of claim 1, wherein the instructions arefurther configured to instruct the at least one processor to: identify,based on braking events associated with the vehicles, a first location;in response to identifying the first location, analyze image dataassociated with the first location, the analyzing comprising performingpattern recognition on image data to determine at least one objectassociated with the first location.
 8. The apparatus of claim 1, whereinthe instructions are further configured to instruct the at least oneprocessor to: receive data regarding braking events, each eventoccurring on one of the autonomous vehicles, and each event associatedwith a location; determine that the braking events correspond to apattern; identify, based on determining that the braking eventscorrespond to the pattern, a first location; wherein determining thatthe first vehicle is failing to properly detect objects comprisesdetermining that the first vehicle is failing to properly detect objectsat the identified first location.
 9. The apparatus of claim 8, whereindetermining that the braking events correspond to the pattern comprisesidentifying the pattern based at least in part on an output from amachine learning model.
 10. An apparatus comprising: at least oneprocessor; and memory storing instructions configured to instruct the atleast one processor to: receive data regarding braking events, eachevent occurring on one of a plurality of vehicles, and each eventassociated with a location; identify, based on the received dataregarding the braking events, a first location; determine that a firstvehicle of the vehicles is failing to properly detect objects at thefirst location; and in response to determining that the first vehicle isfailing to properly detect the objects, control a navigation system ofthe first vehicle.
 11. The apparatus of claim 10, wherein theinstructions are further configured to instruct the at least oneprocessor to: store a map based on data regarding braking eventsreceived from the vehicles; and provide, based on the map, a navigationservice to the first vehicle.
 12. The apparatus of claim 10, whereinidentifying the first location comprises determining that the respectivelocations for the braking events are within a predetermined area. 13.The apparatus of claim 10, wherein identifying the first locationcomprises comparing a deceleration of the respective vehicle for each ofthe braking events to a predetermined threshold.
 14. The apparatus ofclaim 10, wherein identifying the first location comprises at least oneof: determining that the respective location for each of the brakingevents is within a predetermined distance of the first location; ordetermining that a distance between the respective locations for thebraking events is within a predetermined value.
 15. The apparatus ofclaim 10, wherein each of the braking events corresponds to activationof at least one of an automatic emergency braking system or an anti-lockbraking system of the respective vehicle.
 16. The apparatus of claim 10,wherein identifying the first location comprises comparing a measurementassociated with activation of a braking system of the respective vehicleto a predetermined threshold.
 17. The apparatus of claim 16, wherein themeasurement is associated with movement of a brake pedal by a user ofthe respective vehicle.
 18. The apparatus of claim 17, wherein thepredetermined threshold is a level of pressure or an extent of motionassociated with depressing of the brake pedal.
 19. The apparatus ofclaim 10, wherein identifying the first location comprises comparing anumber of the braking events that occur within a predetermined timeperiod to a threshold.
 20. The apparatus of claim 10, wherein thecommunication further causes the first vehicle to control an operationof the first vehicle based on an output from an artificial neuralnetwork.
 21. A method comprising: receiving data regarding brakingevents, each event occurring on one of a plurality of autonomousvehicles, and each event associated with a location; identifying, basedon the received data regarding the braking events, a first location;determining that a first vehicle of the vehicles is failing to properlydetect objects at the first location; and in response to determiningthat the first vehicle is failing to properly detect the objects,deactivating an autonomous driving mode of the first vehicle.
 22. Themethod of claim 21, wherein the objects include at least one firstobject, and wherein the first vehicle stores an artificial neuralnetwork (ANN) model used to detect the first object when operating inthe autonomous driving mode.